Part replacement predictions using convolutional neural networks

ABSTRACT

An example of an apparatus including a communication interface to receive an image file is provided. The image file represents a scanned image of a output generated by a printing device. The apparatus further includes an identification engine to process the image file with a convolutional neural network model to identify a feature. The feature may be indicative of a potential failure. The apparatus also includes an image analysis engine to indicate a life expectancy of a part associated with the potential failure based on the feature. The image analysis engine uses the convolutional neural network model to determine life expectancy.

BACKGROUND

Various devices and apparatus have parts or components with anundetermined life expectancy. The parts or components may failperiodically leading to the parts or components to be replaced.Component failure may not be a complete failure and instead lead to adecrease in the performance of the device or apparatus. Accordingly, thedecrease in performance is to be diagnosed in order to identify aspecific part or component that is to be serviced or replaced.

BRIEF DESCRIPTION OF THE DRAWINGS

Reference will now be made, by way of example only, to the accompanyingdrawings in which:

FIG. 1 is a block diagram of an example apparatus to monitor parts of aprinting device;

FIG. 2 is a flowchart of an example of a method of monitoring parts of aprinting device;

FIG. 3 is a block diagram of another example apparatus to monitor partsof a printing device;

FIG. 4 is a block diagram of another example apparatus to monitor partsof a printing device with a scanning module; and

FIG. 5 is an example of a printing device with the apparatus of FIG. 4 .

DETAILED DESCRIPTION

Output from printing devices may be widely accepted and may often bemore convenient to use. In particular, output from printing devices areeasy to distribute, store, and be used as a medium for disseminatinginformation. In addition, output from printing devices may serve ascontingency for electronically stored data which is to be presented byan electronic device, such as may happen when the electronic devicefails, such as with a poor data connection for downloading the data fileand/or a depleted power source.

With repeated use of a printing device to generate output, the printingdevices use various parts or components that may wear down over time andeventually fail. Failure of a part in a printing device may result insignificant down time to diagnose the problem to identify the part. Inaddition, if the failed part is not known, repair and replacement of thepart is not possible. To reduce the amount of downtime of a printingdevice, some parts have estimated life expectancies measured in time,usage, or a combination of both. Accordingly, parts may be preemptivelyreplaced to reduce the likelihood of downtime affecting the printingdevice. In order to prevent downtime, the estimated life expectancies ofparts may be reduced to decrease the probability of a premature failure.Even with the reduced estimated life expectancies, parts may fail beforeits estimated life expectancy. This may cause the printing device to goout of service during the diagnosis and repair or replacement the failedpart.

In some instances, the quality of the output from printing devices mayprovide some indication as to the health and/or life expectancy of someparts. Since there are a large number of parts in a printing, failure ofa single part may result in a small change to the quality of the outputfrom the printing device. In some instances, the change to the qualityof the output from the printing device may not be noticeable to humanoperator. In other instances, the change to the quality of the outputfrom the printing device may result in a noticeably imperfect output,which may in turn prompt an operator to file a complaint regarding theprinting device quality. Such complaints are difficult to diagnose sinceone of many parts may be the cause. However, using machine learningmethods to review the quality of the output from the printing device mayprovide a quick diagnosis that may reduce downtime for a printingdevice.

Referring to FIG. 1 , an example apparatus to monitor parts of aprinting device is generally shown at 10. The apparatus 10 may includeadditional components, such as various memory storage units, interfacesto communicate with other computer apparatus or devices, and furtherinput and output devices to interact with the user. In addition, inputand output peripherals may be used to train or configure the apparatus10 as described in greater detail below. In the present example, theapparatus 10 includes a communication interface 15, an image analysisengine 20, and an identification engine 25. Although the present exampleshows the image analysis engine 20 and the identification engine 25 asseparate components, in other examples, the image analysis engine 20 andthe identification engine 25 may be part of the same physical componentsuch as a microprocessor configured to carry out multiple functions.

The communications interface 15 is to receive an image file. In thepresent example, the source of the image file is not particularlylimited. For example, the image file may be generated at a printingdevice with the generation of output. In other examples, the image filemay be generated using a scanning device. The image file may represent ascanned image of output generated by the printing device. For example,the output may be a document, a photograph, or a three-dimensionalobject. In the present example, the communications interface 15 mayreceive signals from a plurality of printing devices, such as if theapparatus 10 were to be operated as a service provided over a cloudnetwork. Accordingly, the communications interface 15 may be to receiveother data associated with the printing device, such as an identifier ofthe specific printing device, or data describing specifics of eachprinting device, such as a model number so that specifications of theprinting device may be retrieved from a database.

The manner by which the communications interface 15 receives the imagefile from a printing device is not limited and may include receiving anelectrical signal via a wired connection. For example, thecommunications interface 15 may be connected to the printing device inexamples were the apparatus 10 is part of the printing device, such aspart of an onboard diagnosis system. In other examples, thecommunications interface 15 may receive wireless signals such as via aBluetooth connection, radio signals or infrared signals from thescanning device. In further examples, the communications interface 15may be a network interface for communicating over a local area networkor the Internet where the communications interface 15 may receive datafrom a remote printing device.

The identification engine 20 is to process the image file received by atthe communications interface 15. In particular, the identificationengine is to apply a convolutional neural network model to identify afeature in the image file. The specific feature to be identified fromthe image file is not particularly limited. In the present example, thefeature may include a defect that is indicative of a potential failureor underperforming part.

The manner by which the convolutional neural network model is applied isnot limited. It is to be appreciated that a convolutional neural networkmodel uses multiple layers to identify features in an image. This modelmay be used on images with no preprocessing such that a raw imagereceived by the communications interface 15 may be directly processed bythe identification engine 20. For example, the convolutional neuralnetwork model may be a four layer convolutional neural network modeltrained with a training dataset having 64,000 training images anddividing each image file into a 128×128 grid. In other examples, theconvolutional neural network may be a pre-built image recognition modelthat has been retrained with new fully connected layers and an outputlayer.

In the present example, the identification engine 20 is to use theconvolutional neural network model trained using various training datadesigned to identify a feature of an image file on which furtheranalysis is to be carried out. The feature may be a specific portion ofthe image file, or it may be a characteristic applied over the entireimage file which is unlikely to be part of the intended image.

In the present example, the feature indicative of a potential failuremay be a type of defect or a combination of defects observed in aportion of the image file. For example, the convolutional neural networkmodel may be able to identify a banding issue where uniform portions ofthe image appear to have bands of slightly different shades. Anotherfeature in the image file that may be identified may be an issue withthe color plane registration causing portions of the image to appear outof focus, such that sharp edges may appear fuzzy. Furthermore, anotherfeature or defect that may be identified by the identification engine 20may be a fade where the image loses contrast or color intensity incertain portions. Other features or defects in an image file may includeghosting, streaking, shotgun, spots, or other visible image defects.

In other examples, the image file may represent a three-dimensionalobject obtained using a three-dimensional scanner of a printed object.

The image analysis engine 25 is to identify a part of a printing deviceassociated with the feature identified by the identification engine 20.For example, the image analysis engine 25 may associate a part of aprinting device and indicate a life expectancy based on the feature, ora combination of features, and the prominence of each feature identifiedby the identification engine 20. In the present example, the imageanalysis engine 25 uses a convolutional neural network model todetermine the life expectancy of the part.

The manner by which the convolutional neural network model is applied isnot limited. In the present example, the same convolutional neuralnetwork model that was applied to by the identification engine 20 isused by the image analysis engine 25. In other examples, separateconvolutional neural network model trained with a separate trainingdataset may be applied.

Although the present examples use a convolutional neural network modelto analyze the images, other artificial intelligence or machine learningmodels may be used. For example, other model may include support vectormachines, random forest trees, Naïve Bayes classifiers, recurring neuralnetworks, and other types of neural networks.

Referring to FIG. 2 , a flowchart of an example method of monitoringparts of a printing device is generally shown at 200. In order to assistin the explanation of method 200, it will be assumed that method 200 maybe performed with the apparatus 10. Indeed, the method 200 may be oneway in which apparatus 10 may be configured. Furthermore, the followingdiscussion of method 200 may lead to a further understanding of theapparatus 10 and its various parts. In addition, it is to be emphasized,that method 200 may not be performed in the exact sequence as shown, andvarious blocks may be performed in parallel rather than in sequence, orin a different sequence altogether.

Beginning at block 210, a scanned image of a document or other outputfrom a printing device is received at the apparatus 10. In the presentexample, the scanned image may be received by a communications interface15. The manner by which the scanned image is received is not limited andmay involve receiving an image file from a printing device via a wiredconnection or a wireless connection such as via a Bluetooth connection,radio signals or infrared signals. For example, the image file may be ina standard format such as JPEG, TIFF, GIF, PNG, BMP, or other formats.In other examples, the image file may be in a non-standard proprietaryformat associated with the printing device. In some examples, thescanned image may be received directly from an onboard diagnosis systemof a printing device. In other examples, the scanned image may bereceived via a network interface for communicating over a local areanetwork or the Internet where the image files may be received from aremote printing device.

Block 220 uses the identification engine 20 to identify a defect in thescanned image. In the present example, the identification engine 20applies a convolutional neural network model to process the scannedimage received at block 210. The defect to be identified from thescanned image is not particularly limited. In the present example, thedefect may be an indication of a potential failure or underperformingpart.

In the present example, the convolutional neural network model usesmultiple layers to identify features in an image. The convolutionalneural network model may be used on images with little or nopreprocessing such that a raw image received in block 210 may bedirectly processed. In the present example, the convolutional neuralnetwork model may be a four layer convolutional neural network modeltrained with a training dataset having 64,000 training images anddividing each image file into a 128×128 grid. For example, an existingimage recognition model, such as VGG16, may be used. In one example, therecognition model may be re-trained using a prior fully connected layerand output layer to successfully leverage the feature extraction fromexisting models. In some examples, VGG16 may use input image sizes ofabout 224×224. In other examples, the training set may be expandedbeyond 64,000 images to about 100,000 to about 1,000,000 images.

In some examples, a rendered image from a data file may be used to trainthe convolutional neural network model as applied by the identificationengine. In such an example, the rendered image may be generated from asource file sent to a printing device. Accordingly, the rendered imagemay be considered to be a passing sample with no defects. It is to beappreciated that the rendered image and the scanned image maysubsequently be added to a database of images as part of a trainingdataset. Therefore, with each scanned image and rendered image, theapparatus 10 builds a larger training dataset to further increase theaccuracy of identifying defects. The defects that may be identified bythe identification engine 20 carrying out block 220 is not limited andmay include defects such as banding, color plane registration issues,fading, ghosting, streaking, shotgun, and spots.

Block 230 involves determining a part associated with the defectidentified in block 220. The manner by which the part associated withthe defect is identified is not particularly limited. For example, thepart may be identified using the same convolutional neural network modelused in block 220. In other examples, a different convolutional neuralnetwork model or other machine learning model may be used. In furtherexamples, other methods may be used to associate the part with thedefect, such as lookup tables. In further examples, telemetry data mayalso be used by the identification engine to assist in theidentification of a part causing a defect or a plurality of defects.

Block 240 determines the life expectancy of the part identified in block230. In the present example, the image analysis engine 25 applies aconvolutional neural network model to process the scanned image receivedat block 210 in view of the classifications made at block 220 and block230. The life expectancy of the part is then to be determined.

In the present example, once the part has been identified and the lifeexpectancy determined, it is to be appreciated that the information maybe provided to a service provider of the printing device. For example,for smart printing devices, a message may be sent to a service providerindicating that a specific part may be expected to fail within a certainperiod of time. This may allow the service provider to proactively offerservices or part replacements to the operator of the printing device. Inother examples, the printing device may display the results of block 240on a display of the printing device for an operator to view.

It is to be appreciated that variations are contemplated. For example,blocks 220, 230, and 240 may be carried out at the same time in a onestep processes to determine a part life expectancy directly fromprocessing the image. In other words, the convolutional neural networkmay operate directly on the image without identifying any defects in theimage.

Referring to FIG. 3 , another example of an apparatus to monitor partsof a printing device is shown at 10 a. Like components of the apparatus10 a bear like reference to their counterparts in the apparatus 10,except followed by the suffix “a”. The apparatus 10 a includes acommunication interface 15 a, a processor 30 a to execute an imageanalysis engine 20 a and an identification engine 25 a, a renderingengine 35 a, and a memory storage unit 40 a.

The communications interface 15 a is to receive an image file. In thepresent example, the source of the image file is not particularlylimited. For example, the image file may be generated at a printingdevice with the generation of output. In other examples, the image filemay be generated using a scanning device. The image file may represent ascanned image of output generated by the printing device, such as, theoutput may be a document, a photograph, or a three-dimensional object.In addition, the communications interface 15 a may also receive a sourcefile for the rendering engine 35 b. In the present example, the sourcefile may be received from the printing device or it may be received froma print server that is to send the source file to the printing device togenerate output.

The processor 30 a may include a central processing unit (CPU), agraphics processing unit (GPU), a microcontroller, a microprocessor, aprocessing core, a field-programmable gate array (FPGA), anapplication-specific integrated circuit (ASIC), or similar. Theprocessor 30 a and the memory storage unit 40 a may cooperate to executevarious instructions. The processor 30 a may execute instructions storedon the memory storage unit 40 a to carry out processes such as themethod 200. In other examples, the processor 30 a may executeinstructions stored on the memory storage unit 40 a to implement theidentification engine 20 a and/or the image analysis engine 25 a. Inother examples, the identification engine 20 a and/or the image analysisengine 25 a may each be executed on a separate processor. In furtherexamples, the identification engine 20 a and/or the image analysisengine 25 a may be operated on a separate machine, such as from asoftware as a service provider.

The identification engine 20 a and the image analysis engine 25 a arenot particularly limited and function similar to the identificationengine 20 and the image analysis engine 25 as described above,respectively. In particular, the identification engine 20 a is toprocess the image file received by at the communications interface 15 aby applying a convolutional neural network model to identify a featurein an image file. In the present example, the identification engine mayidentify the feature based on differences between a rendered imagegenerated by the rendering engine 35 a as discussed in greater detailbelow, and the image file received by the communications interface 15 a.

The image analysis engine 25 a is to identify a part of a printingdevice associated with the feature identified by the identificationengine 20 a. In addition, the image analysis engine 25 a may indicate alife expectancy of the part based on the feature and the prominence ofthe feature identified by the identification engine 20 a using theconvolutional neural network model.

The rendering engine 35 a is to generate a rendered image file from asource file. In the present example, the source file may be received bya print server for distribution to a printing device. The source filerepresent data which the printing device may use to generate output. Forexample, the output generated may be a printed document based on asource file generated by an image editing program on a personalcomputer. Continuing with this example, the rendering engine 35 a mayuse the source file to render an image file to represent the intendedimage to be generated by the printing device and may be considered to bea passing sample with no defects. The passing sample may then be used totrain the convolutional neural network model used by the identificationengine 20 a to identify features or defects. The manner by which animage file is generated from a source file is not particularly limited.For example, the rendered image file may be visual representation of adata file which may involve rasterization or casting of the source file.

The memory storage unit 40 a is coupled to the processor 30 a and therendering engine 35 a and may include a non-transitory machine-readablestorage medium that may be any electronic, magnetic, optical, or otherphysical storage device. In the present example, the memory storage unit40 a may store an operating system 100 a that is executable by theprocessor 30 a to provide general functionality to the apparatus 10 a,for example, functionality to support various applications. Examples ofoperating systems include Windows™, macOS™, iOS™, Android™, Linux™, andUnix™. The memory storage unit 40 a may additionally store instructionsto operate the identification engine 20 a and the image analysis engine25 a at the driver level as well as other hardware drivers tocommunicate with other components and other peripheral devices of theapparatus 10 a, such as the communications interface 15 a or variousoutput and input devices (not shown).

In the present example, the memory storage unit 40 a may also maintainan image database 105 a to store image files received via thecommunication interface 15 a. In the present example, the memory storageunit 40 a may receive a plurality of image files to store in the imagedatabase 105 a where the image files may be subsequently processed. Forexample, the apparatus 10 a may receive multiple image files to processand the image files may be stored in a queue in the image database 105 afor processing. In addition, once the image files have processed, thememory storage unit 40 a may retain the image files in the imagedatabase 105 a for subsequent use for training purpose of theconvolutional neural network model.

The memory storage unit 40 a may also maintain a training dataset 110 ato training data for training the convolutional neural network model.The training dataset 110 a is not particularly limited and may be loadedonto the memory storage unit 40 a from an external source or library. Inother examples, the training dataset 110 a may be developed andaugmented with each operation of the apparatus 10 a to analyze a scannedimage. Furthermore, although a single training dataset 110 a isillustrated, it is to be appreciated that multiple training datasets maybe used to train multiple convolutional neural network models, such asin the case of the identification engine 20 a and the image analysisengine 25 a using different convolutional neural network models.

Referring to FIG. 4 , another example of an apparatus to monitor partsof a printing device is shown at 10 b. Like components of the apparatus10 b bear like reference to their counterparts in the apparatus 10 a,except followed by the suffix “b”. The apparatus 10 b includes acommunication interface 15 b, a processor 30 b to execute an imageanalysis engine 20 b and an identification engine 25 b, a renderingengine 35 b, and a memory storage unit 40 b. In the present example, thememory storage unit 40 b also includes an operating system 100 b that isexecutable by the processor 30 b, an image database 105 b, and atraining dataset 110 b.

The apparatus 10 b further includes a scanner module 50 b. In thescanner module is to generate an image file based on an output of aprinting device. In the present example, the scanner module 50 b may bein communication with the communication interface. The scanner module 50b is not particularly limited and may include any device to generate ascanned image based on an object, such as output from a printing device.In the present example, the apparatus 10 b may be a standalone devicefor scanning a variety of different output from multiple printingdevices. Accordingly, the apparatus 10 b may be used by a technician oran operator of printing devices to diagnose the printing device.

In other examples, the apparatus 10 b may be installed on a printingdevice 300 as shown in FIG. 5 . In this example, the apparatus 10 b isinstalled as an inline scanner module configured to scan each documentgenerated by the printing device 300. Furthermore, the apparatus 10 bmay be connected to a network to provide life expectancy data over thenetwork to a central monitoring station. In such an example, theapparatus 10 b maybe part of a smart printer where the parts of theprinter are monitored such that remote diagnosis and solutions toissues, such as print quality issues may be address.

It should be recognized that features and aspects of the variousexamples provided above may be combined into further examples that alsofall within the scope of the present disclosure.

What is claimed is:
 1. An apparatus comprising: a communicationinterface to receive an image file, wherein the image file represents ascanned image of an output generated by a printing device, wherein thescanned image has arbitrary content; an identification engine to processthe image file with a convolutional neural network model to identify afeature, wherein the feature may be indicative of a potential failure,wherein the convolutional neural network was trained with a plurality oftraining images to identify the feature; and an image analysis engine toindicate a life expectancy of a part associated with the potentialfailure based on the feature, wherein the image analysis engine uses theconvolutional neural network model to determine life expectancy.
 2. Theapparatus of claim 1, further comprising a rendering engine.
 3. Theapparatus of claim 2, wherein the rendering engine is to generate arendered image file.
 4. The apparatus of claim 3, wherein the renderedimage file is a passing sample to train the identification engine. 5.The apparatus of claim 4, wherein the identification engine is toidentify the feature based on a difference between the image file andthe rendered image file.
 6. The apparatus of claim 1, further comprisinga memory storage unit to store a training dataset, wherein the imageanalysis engine uses the training dataset to process the image file. 7.The apparatus of claim 1, further comprising a scanner module incommunication with the communication interface, wherein the scannermodule is to generate the image file.
 8. A method comprising: receivinga scanned image of a document of arbitrary content, wherein the documentincludes a defect; identifying the defect with an identification engine,wherein the identification engine applies a convolutional neural networkmodel to process the scanned image, wherein the convolutional neuralnetwork was trained with a plurality of training images to identifyimage features indicative of defects; analyzing the defect to determinea part associated with the defect; and determining a life expectancy ofa part associated with the defect, wherein determining the lifeexpectancy uses the convolutional neural network model.
 9. The method ofclaim 8, further comprising generating a rendered image.
 10. The methodof claim 9, further comprising using the rendered image as a passingsample to train the identification engine.
 11. The method of claim 10,further comprising scanning output from a printing device with a scannermodule to generate the scanned image.
 12. A non-transitorymachine-readable storage medium encoded with instructions executable bya processor, the non-transitory machine-readable storage mediumcomprising: instructions to receive a scanned image of output generatedby a printing device, wherein the scanned image has arbitrary content;instructions to identify a feature in the scanned image by applicationof a convolutional neural network model to process the scanned image,wherein the convolutional neural network was trained with a plurality oftraining images to identify the feature; instructions to analyze thefeature to determine a part associated with the feature; andinstructions to determine a life expectancy of a part of the printingdevice based on the feature, wherein determining the life expectancyuses the convolutional neural network model.
 13. The non-transitorymachine-readable storage medium of claim 12, further comprisinginstructions to generate a rendered image.
 14. The non-transitorymachine-readable storage medium of claim 13, further comprisinginstructions to apply the convolutional neural network model on thescanned image and the rendered image.
 15. The non-transitorymachine-readable storage medium of claim 14, further comprisinginstructions to scan output from a printing device with a scanner moduleto generate the scanned image.